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Abstract Body 


Background / Context: 

School districts are increasingly adopting technology-based resources in an attempt to improve 
student achievement (Knezek, 2008). Spending on educational hardware is expected to grow 
from $13 billion in 2013 to $19 billion in 2018, an annualized increase of eight percent, while 
districts spend an estimated $8 billion annually on software (Nagel, 2014; Richards & Stebbins, 
2012). A growing body of research has emerged attempting to measure the causal impact of 
technology resources on various student achievement outcomes (Barrow, Markman, & Rouse, 
2008; Campuzano, Dynarski, Agodini, & Rail, 2009; Given, Wasserman, Chari, Beattie, & Eden, 
2008; James-Burdumy et al., 2009; Roschelle et al., 2007; Rouse & Krueger, 2004; R. S. Savage, 
Abrami, Hipps, & Deault, 2009; R. Savage et al., 2013; Wijekumar, Meyer, & Lei, 2012). Yet 
we still know very little about the broad impact of technology resources, as the abundance of 
high quality research ranges across geography, subjects, and grade levels. 

This paper reports the two-year results from randomized control trial of Achieve3000 in the 
Wake County Public School System (WCPSS) in Raleigh, North Carolina. Achieve3000 is an 
early literacy program that differentiates non-fiction reading passages based on individual 
students’ Lexile scores. The driving force behind implementing Achieve3000 in particular, and 
focusing on literacy in general, was the state’s 2012 Read to Achieve legislation, which 
mandated that students failing to meet proficiency standards in grade 3 reading must attend 
summer camp and pass the state’s Read to Achieve test before being promoted to grade 4. 

District staff were eager to launch a new program that could help increase reading proficiency 
among students receiving core instruction — as opposed to at-risk intervention — and ultimately 
reduce the share of children retained in grade 3 as a result of the new legislation. To our 
knowledge, this study constitutes the first randomized control trial of a product from the 
Achieve3000. According to its website, Achieve3000 has existed for more than a decade, serves 
more than 1 million U.S. students, and is consistently ranked by Inc. Magazine as one of the 
fastest growing private education companies in the United States (Magazine, 2014). 

Purpose / Objective / Research Question / Focus of Study: 

According to the Achieve3000 National Lexile study, use of the product “dramatically increases 
students’ Lexile reading gains and builds the critical literacy capacities that are at the heart of the 
Common Core State Standards” (Achieve3000, 2012). Consequently, we address two key 
questions and focal areas for our research: 

• Do students who use Achieve3000 outperform students who do not use Achieve3000 on a 
host of reading achievement measures? 

• Does the performance of students who use Achieve3000 differ across student subgroups? 

Setting: 

The Wake County Public School System is the 15th largest local education agency (LEA) by 
student enrollment in the U.S. and the largest in North Carolina. The system in 2014-15 had 171 
schools educating roughly 155,000 students, approximately half of whom were White, a quarter 
Black, and 15 percent Hispanic/Latino. Approximately one-third qualify for free- or reduced- 
price lunch. As a countywide district, WCPSS has suburban, urban, and rural features allowing 
for a diverse representation of school characteristics. 
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Population / Participants / Subjects: 

32 elementary schools within the school system volunteered to receive Acheive3000 and 
implement the program with fidelity. These 32 schools were then matched on their 2012-13 
elementary school End-of-Grade (EOG) reading composite score, and 16 were randomly selected 
to receive the treatment while the other 16 schools serve as the control group. The sample 
includes nearly 35,000 students in approximately 745 classrooms across grades 2-5. Figure 1, 
below, presents the random assignment process (Moher et al., 2010). 

Intervention / Program / Practice: 

The Achieve3000 suite includes six separate applications, four of which — KidBiz3000, 
TeenBiz3000, Empower3000, and Spark3000 — the company claims are inspired by the work of 
R.C. Anderson on prior knowledge, Carol Ann Tomlinson on differentiation, Michael Kamil on 
the role of technology, and Linda Duncan on vocabulary development. Achieve3000’s theory of 
action is that students will become college and career ready if they are able to read non-fiction 
texts at Lexile levels that exceed 1350. To help students reach that level, the company’s 
applications administer an assessment that establishes a baseline Lexile level. From there, 
students are exposed to non-fiction adaptive reading passages that are aligned to their Lexile 
levels and adjust on the basis of end-of-lesson assessments. This way, reading passages are 
custom tailored to each student’s Lexile level so that students don’t spend valuable instructional 
time reading passages that are either too easy or too difficult (Achieve3000, 2011) . 

Staff from the district and the vendor jointly decided that in order to reach the annual goal of 80 
completed activities, students would utilize KidBiz3000 — the elementary school application — 
twice weekly for 30 minutes. On initial use, students took a 30-minute assessment to obtain a 
baseline Lexile score. For each activity, students would follow a five-step procedure: (1) take a 
poll and respond to it through the KidBiz3000 email application; (2) read a non-fiction article 
aligned with their current Lexile level; (3) complete a series of multiple choice questions; (4) 
vote in a post-reading poll; and (5) answer a “Thought Question.” 

Research Design: 

We employ an experimental research design in the form of a cluster-randomized trial. To 
estimate the impact of Achieve3000 on students who are nested within schools and where 
schools are the unit of assignment, we fit our data to a cluster two-level model with random 
effects where schools were randomized within matched pairs based on a school-level prior 
achievement score, as noted above. We estimate impact on various summative assessments 
controlling for prior achievement and a host of additional student- and school-level controls. In 
measuring the impact of Achieve3000 on students nested in schools, we employ intent-to-treat 
(ITT) and treatment-on-treated (TOT) estimations. The ITT estimation represents the impact of 
the exogenous offer of Achieve3000 to the 16-school sample of treatment sites. This estimation 
represents the empirical estimate of the offer irrespective of the rate of takeup. We investigate 
takeup using instrumental variables estimation. 

Data Collection and Analysis: 

Data for this investigation come from the school system’s administrative and testing records; 
Amplify, Inc.’s mClass reporting system; and Achieve3000’s activity completion and LevelSet 
Lexile pre- and post-test assessments. See Tables 1 and 2, below, for school- and student-level 
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baseline characteristics. We linked all datasets using unique student identification numbers. Our 
model specification is as follows: 

OUTCOME y = p a + P& Y3KOFFER} tJ + Z (j + £ {j + u t 

Where OUTCOME represents our dependent variable of interest (score change or summative 
score for each literacy measure) for student j nested in school i; fio represents the constant; [U 
represents the coefficient of our A3KOFFER predictor (the offer of Achieve3000) for student j 
nested in school i; Z represents a vector of student- and school-level control variables; residual s \j 
represents the random effect of student j in school z; and residual u, represents the random effect 
of school i. 

To estimate the treatment-on-treated impact of using Achive3000 we employ a two-stage least 
square regression where we instrument for actual program use in the first stage. The first stage of 
this model is specified as: 

A3KUSE U =p 0 + P^ASKOFFER) tJ - + Z ijk + E u + u t 

Where A3KUSE represents our dependent variable of interest (Achieve3000 use level) for 
student j nested in school z; fi\ represents the coefficient of our A3KOFFER predictor (the offer 
of Achieve3000) for student j nested in school i. The second stage of this model is specified as: 

OUT COME = P 0 + fi t (JI3KUSE) v + Z tJ - + + Ui 

Where A3KUSE represents the exogenous portion of the offer variable. We instrument for use 
using a conservative implementation level equal to students completing at least one activity. 

Findings / Results: 

Two-year pooled results show that Achieve3000 did not have a significant impact on student 
outcomes. However, both intent-to-treat (ITT) and treatment- on-treated (TOT) estimates show 
that in 2015, the second year of implementation, students in the treatment group outperformed 
their control-group counterparts by 0.13 standard deviation units (a) on the year-end 
Achieve3000 Fexile test (see Tables 3 and 4, below). This effects size is consistent with mean 
empirical effect sizes reported by Fipsey et al. (2012). Yet in neither the pooled nor annual 
samples did Achieve3000 significantly impact student performance on additional Fexile 
outcomes (NC EOG Fexile) or various DIB EES assessments. These aggregate results, in 
conjunction with slightly better implementation in the second year, suggest that the treatment 
group may (1) benefit from improved implementation and (2) may be conditioned, by virtue of 
familiarity with the user interface, to outperform their control group counterparts. 

To address our second research question, we examined interaction effects across a range of 
subgroups. In the pooled sample in both ITT and TOT estimations, male students gained 0.03a 
while in the ITT estimation, students with disabilities gained 0.05a. In the second-year sample, 
both ITT and TOT estimates show gains of 0.05a for male students, 0.09a for students with 
disabilities, 0.05a for gifted students, and 0.14a for students attending year-round schools. 


SREE Spring 2016 Conference Abstract Template 


A- 3 


We also address implementation since, as the program’s theory of action suggests, greater 
impacts should correspond with increased use of the program. The program’s developers group 
activity completion into three main categories: completion of 1-39 activities, 40-79 activities, 
and 80 or more activities. The percentage of students completing activities in each of these three 
categories increased from 2013-14 to 2014-15. Figure 3, below, shows that the percentage of 
students completing 80 or more activities nearly doubled from 5.5% to 9.3%. In an extension of 
our implementation analysis, we plan to derive matched control groups from treatment group 
profiles according to implementation category and compare student achievement outcomes 
across these groups. Early work across the pooled sample and by year suggests that determinants 
of activity completion are primarily being male (2 additional activities), attending a year-round 
school (6-8 activities), or attending a Title I school (1-7 activities). As expected, large and 
significant cohort effects (5 activities) emerge in the pooled sample (see Table 5, below). 

Conclusions: 

Results from a two-year randomized control trial of Achieve3000 suggest that the impacts, 
implementation, and overall promise of a technology literacy solution fell short of expectations. 
This comes as niche technology solutions like Achieve3000 increasingly complement and even 
supplement non-digital literacy instruction in schools and classrooms across the country. Yet 
moderate effects may arrive with improved implementation, as our second- year results suggest. 
While we did not detect an impact on our primary Lexile outcome in our pooled sample, the 
effect size in the second year (0.13a) was consistent with empirical averages. On the same 
outcome in year two, we detected treatment effects for students who are male, have a disability, 
are gifted, or attend year-round schools. However, we could not triangulate these results with 
similar outcome measures, suggesting that implementation and program familiarity (i.e., the 
Lexile test is embedded in the software) may explain our second- year treatment effects. 

We additionally conclude that implementation remains a challenge in treatment schools. Despite 
the fact that our analytic sample of 32 schools all confirmed that they would implement 
Acheive3000 with fidelity if selected to receive the program, usage data shows that two-thirds of 
students in the treatment sample completed fewer than 40 activities in the second year of 
implementation (see Figure 3). This result stands in stark contrast to the vendor’s full 
implementation goal of all students completing 80 or more activities. 

The consequences for school districts in general and WCPSS in particular are two-fold. First, 
district stakeholders should adopt new instructional tools with reasonable expectations about 
how they might impact student achievement. Our results, along with reviews of similar RCTs 
(Lipsey et al., 2012), broad literacy interventions (Kim & Quinn, 2013) and technology tools 
(Takacs, Swart, & Bus, 2015), suggest that empirical effects are much smaller than theoretical 
ones. Second, districts need to develop strict implementation plans that include consistent 
monitoring, central office-to-school feedback loops, and accountability for weak implementation. 

Study limitations are mostly implementation-based and include: limited generalizability on the 
basis of full implementation, select system features that did impact implementation (e.g., 
program start date; technology challenges), and staffing changes at the district- and vendor-level 
that may have impacted implementation. We will add attrition analyses to our impact and 
implementation findings. 
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Appendix B. Tables and Figures 


Figure 1: Assignment of Achieve3000 
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n=32 WCPSS elementary 
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Allocated to Control condition: n=16 
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refused to administer pre-test 
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Table 1: School-Level Baseline Characteristics 



Control 

Treatment 

Control - 

Treatment 

p-value from joint 
orthogonality test of 

treatment arms 

School characteristics 

School % Male 

0.520 

0.517 

0.002 

0.682 


(0.004) 

(0.004) 

(0.006) 


School % ED 

0.368 

0.356 

0.012 

0.824 


(0.041) 

(0.032) 

(0.052) 


School % SWD 

0.116 

0.112 

0.004 

0.703 


(0.006) 

(0.007) 

(0.009) 


School % LEP 

0.092 

0.124 

-0.032 

0.122 


(0.013) 

(0.016) 

(0.020) 


School % AIG 

0.043 

0.041 

0.002 

0.873 


(0.009) 

(0.005) 

(0.010) 


School % Hispanic 

0.176 

0.215 

-0.039 

0.238 


(0.022) 

(0.023) 

(0.032) 


School % Black 

0.309 

0.203 

0.106** 

0.045 


(0.043) 

(0.027) 

(0.051) 


Prior achievement 

Achieve3000 Pretest 

-0.083 

0.006 

-0.089 

0.391 


(0.084) 

(0.059) 

(0.103) 


EOG Lexile Pretest 

-0.012 

-0.028 

0.016 

0.869 


(0.075) 

(0.059) 

(0.095) 


DIBELS Composite Pretest 

-0.028 

-0.008 

-0.020 

0.829 


(0.079) 

(0.050) 

(0.093) 


DIBELS Oral Reading Fluency Pretest 

-0.043 

-0.004 

-0.038 

0.681 


(0.077) 

(0.052) 

(0.093) 


N 

16 

16 

32 


Proportion 

0.500 

0.500 




* p< 0.1; ** p<0.05; *** p<0.01 
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Table 2: Student-Level Baseline Characteristics 



Control 

Treatment 

Control - 

Treatment 

p-value from joint 
orthogonality test 
of treatment arms 

Student characteristics 

Cohort 

0.521 

0.516 

0.005 

0.390 


(0.004) 

(0.004) 

(0.005) 


Male 

0.511 

0.510 

0.001 

0.863 


(0.004) 

(0.004) 

(0.005) 


Black 

0.302 

0.208 

q 094*** 

0.000 


(0.004) 

(0.003) 

(0.005) 


Hispanic 

0.172 

0.195 

-0.023*** 

0.000 


(0.003) 

(0.003) 

(0.004) 


LEP 

0.083 

0.102 

-0.019*** 

0.000 


(0.002) 

(0.002) 

(0.003) 


SWD 

0.114 

0.112 

0.002 

0.588 


(0.002) 

(0.002) 

(0.003) 


Economically Disadvantaged 

0.359 

0.348 

0.011** 

0.035 


(0.004) 

(0.004) 

(0.005) 


AIG: Reading & Math 

0.071 

0.065 

0.006** 

0.030 


(0.002) 

(0.002) 

(0.003) 


Magnet School 

0.210 

0.191 

0.019*** 

0.000 


(0.003) 

(0.003) 

(0.004) 


Year-Round Calendar 

0.480 

0.583 

-0.103*** 

0.000 


(0.004) 

(0.004) 

(0.005) 


Title 1 

0.604 

0.643 

-0.039*** 

0.000 


(0.004) 

(0.004) 

(0.005) 


Prior achievement 

Achieve3000 Pretest 

-0.035 

0.033 

-0.067*** 

0.000 


(0.009) 

(0.009) 

(0.013) 


EOG Lexile Pretest 

0.006 

-0.005 

0.011 

0.533 


(0.013) 

(0.012) 

(0.018) 


DIBELS Composite Pretest 

-0.006 

0.006 

-0.012 

0.318 


(0.009) 

(0.008) 

(0.012) 


DIBELS Oral Reading Fluency Pretest 

-0.020 

0.018 

-0.038*** 

0.001 


(0.008) 

(0.008) 

(0.012) 


N 

16619 

18013 

34632 


Proportion 

0.480 

0.520 




* p< 0.1; ** p<0.05; *** p<0.01 
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Table 3: Achieve3000 ITT Impact on Achieve3000 Posttest 



2014 

2015 

Pooled 

Achieve3000 

-0.054** 

0 129*** 

0.029 


(0.022) 

(0.033) 

(0.024) 

Pretest 

0.769*** 

0.866*** 

0.821*** 


(0.006) 

(0.004) 

(0.003) 

Cohort 

0.000 

0.000 

0.027*** 


(0.000) 

(0.000) 

(0.006) 

Male 

-0.006 

0.016*** 

0.006 


(0.009) 

(0.006) 

(0.005) 

Black 

-0 H9*** 

-0.084*** 

-0.099*** 


(0.013) 

(0.009) 

(0.008) 

Hispanic 

-0.045*** 

-0.042*** 

-0.045*** 


(0.015) 

(0.010) 

(0.009) 

LEP 

-0.166*** 

-0.068*** 

-0.111*** 


(0.019) 

(0.012) 

(0.011) 

SWD 

-0.245*** 

-0.122*** 

-0.180*** 


(0.015) 

(0.011) 

(0.009) 

ED 

-0.126*** 

-0.069*** 

-0.098*** 


(0.012) 

(0.008) 

(0.007) 

AIG 

0.318*** 

0.158*** 

0.242*** 


(0.017) 

(0.013) 

(0.010) 

Magnet 

0.040 

0.024 

0.038 


(0.031) 

(0.041) 

(0.029) 

Year-Round 

0.029 

-0.010 

0.001 


(0.022) 

(0.033) 

(0.023) 

Title 1 

0.019 

0.033 

0.023 


(0.029) 

(0.042) 

(0.030) 

School % Male 

0.791 

1.674* 

1.282* 


(0.862) 

(0.979) 

(0.698) 

School % ED 

0.435 

-0.359 

-0.082 


(0.313) 

(0.463) 

(0.326) 

School % SWD 

-0.334 

-0.662 

-0.404 


(0.481) 

(0.607) 

(0.430) 

School % LEP 

1.044** 

0.789 

1.063** 


(0.492) 

(0.734) 

(0.518) 

School % AIG 

0.257 

0.129 

0.071 


(0.705) 

(1.038) 

(0.732) 

School % Hispanic 

-1.395*** 

-0.197 

-0.617 


(0.411) 

(0.597) 

(0.421) 

School % Black 

-0.417** 

0.099 

-0.177 


(0.181) 

(0.284) 

(0.200) 

Constant 

-0.201 

-0.784 

-0.535 


(0.410) 

(0.496) 

(0.353) 

Li 

0.038*** 

0.072*** 

0.050*** 


(0.007) 

(0.009) 

(0.007) 

e 

0.443*** 

0.344*** 

0.397*** 


(0.003) 

(0.002) 

(0.002) 

N 

9,732 

12,851 

22,583 


* p< 0.1; ** p<0.05; *** p<0.01 
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Table 4: Achieve3000 TOT Impact on Achieve3000 Posttest 



2014 

2015 

Pooled 

Achieve3000 

-0.064* 

0 131*** 

0.031 


(0.037) 

(0.045) 

(0.035) 

Pretest 

0.769*** 

0.866*** 

0.821*** 


(0.006) 

(0.004) 

(0.003) 

Cohort 

0.000 

0.000 

0.029*** 


(0.000) 

(0.000) 

(0.006) 

Male 

-0.006 

0.016*** 

0.006 


(0.009) 

(0.006) 

(0.005) 

Black 

-0 119*** 

-0.084*** 

-0.099*** 


(0.013) 

(0.009) 

(0.008) 

Hispanic 

-0.045*** 

-0.041*** 

-0.045*** 


(0.015) 

(0.010) 

(0.009) 

LEP 

-0.167*** 

-0.067*** 

-0.111*** 


(0.019) 

(0.012) 

(0.011) 

SWD 

-0.246*** 

-0 121*** 

-0.180*** 


(0.015) 

(0.011) 

(0.009) 

ED 

-0 125*** 

-0.069*** 

-0.098*** 


(0.012) 

(0.008) 

(0.007) 

AIG 

0.318*** 

0.158*** 

0.242*** 


(0.017) 

(0.013) 

(0.011) 

Magnet 

0.043 

0.031 

0.039 


(0.045) 

(0.053) 

(0.040) 

Year-Round 

0.031 

-0.010 

0.000 


(0.032) 

(0.043) 

(0.032) 

Title 1 

0.028 

0.033 

0.022 


(0.042) 

(0.055) 

(0.041) 

School % Male 

0.573 

1.633 

1.308 


(1.232) 

(1.276) 

(0.964) 

School % ED 

0.374 

-0.353 

-0.094 


(0.448) 

(0.606) 

(0.453) 

School % SWD 

-0.202 

-0.641 

-0.405 


(0.692) 

(0.793) 

(0.596) 

School % LEP 

1.078 

0.778 

1.077 


(0.693) 

(0.963) 

(0.718) 

School % AIG 

0.037 

0.164 

0.086 


(1.028) 

(1.360) 

(1.026) 

School % Hispanic 

-1.408** 

-0.163 

-0.600 


(0.579) 

(0.784) 

(0.588) 

School % Black 

-0.432* 

0.090 

-0.167 


(0.262) 

(0.370) 

(0.280) 

Constant 

-0.072 

-0.775 

-0.554 


(0.592) 

(0.646) 

(0.490) 

N 

9,732 

12,851 

22,583 


* p< 0.1; ** p<0.05; *** p<0.01 
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Figure 2: Achieve3000 Activity Completion, Pooled Sample 
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Figure 3: Achieve3000 Activity Completion, by Year 
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Table 5: Determinants of Achieve3000 Activities Completed 



2014 

2015 

Pooled 

Cohort 

0.000 

0.000 

4.709*** 


(0.000) 

(0.000) 

(0.336) 

Male 

2.104*** 

2.214*** 

2.176*** 


(0.428) 

(0.520) 

(0.339) 

Black 

-3.154*** 

-7.156*** 

-5.205*** 


(0.575) 

(0.738) 

(0.472) 

Hispanic 

-3.552*** 

-5.132*** 

-4.320*** 


(0.753) 

(0.924) 

(0.602) 

LEP 

1.712 

1.514 

1.532** 


(1.047) 

(1.143) 

(0.777) 

SWD 

-3.690*** 

-2.226*** 

-2.992*** 


(0.581) 

(0.853) 

(0.512) 

ED 

0.553 

0.628 

0.613 


(0.544) 

(0.691) 

(0.443) 

AIG 

-1.789** 

1.488 

-0.296 


(0.757) 

(1.029) 

(0.629) 

Magnet 

-3.213*** 

1.629** 

-0.774 


(0.598) 

(0.787) 

(0.496) 

Year-Round 

6.460*** 

8.145*** 

7.305*** 


(0.509) 

(0.565) 

(0.381) 

Title 1 

0.878** 

7.032*** 

3.997*** 


(0.432) 

(0.547) 

(0.350) 

Constant 

10.467*** 

10.317*** 

7.989*** 


(0.526) 

(0.580) 

(0.438) 

R 2 

0.03 

0.03 

0.03 

N 

15,162 

15,693 

30,855 


* p<0. 1 ; **p<0.05; ***p<0.01 
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